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PROBLEM 

The  Single  Integrated  Air  Picture  (SIAP)  System  Engineering  Task  Force  (SE  TF) 
requires  the  capability  to  maintain  and  provide  system  interoperability  information  to 
support  joint  interoperability  planning,  testing,  and  forensics  engineering.  This 
Technical  Report  (TR)  describes  and  discusses  the  SIAP  SE  TF’s  development  of  a 
central  Lessons  Learned  Knowledge  Base  (LLKB)  to  address  this  requirement. 


OBJECTIVES 

This  TR  outlines  the  development  process,  procedures,  and  technical  capabilities  of  the 
LLKB.  It  addresses  the  process  to  develop  and  assess  technical  requirements  for  an 
initial  desktop  prototype  and  an  unclassified  on-line  website,  culminating  in  the  delivery 
of  a  Secret  Internet  Protocol  Router  Network  (SIPRN)-based  LLKB  capability.  It 
outlines  the  plan  of  action  followed  to  implement  this  capability  to  support  the  SIAP 
Analysis  Team  (SAT)  and  the  greater  SIAP  community. 


APPROACH 

The  SIAP  Lessons  Learned  Working  Group  (WG)  initiated  prototype  development  as  a 
consequence  of  extensive  liaison  with  joint  and  service  lessons  learned  database 
subject  matter  experts  (SMEs).  Feedback  from  these  SMEs  suggested  that  an  initial 
stand-alone,  PC-based  database  would  be  the  most  effective  first  step,  mitigating  cost, 
schedule  and  technical  concerns.  The  WG  completed  Phase  1  on  schedule  and 
delivered  the  stand-alone  desktop  application  as  the  initial  LLKB  capability.  Phase  2 
encompassed  transition  of  the  prototype  to  an  unclassified  website  to  solicit  user  review 
and  inputs  to  be  incorporated  into  the  design  of  the  objective  LLKB  system.  Feedback 
from  SIAP  TF,  Service  and  Agency  representatives  served  as  the  foundation  for  the 
classified  LLKB  capability.  Phase  3  completed  the  migration  of  the  Phase  2  website  to 
the  SIPRN  and  delivered  a  classified  SIAP  lessons  learned  and  knowledge  repository. 
Phase  4  elicited  further  feedback  added  additional  capabilities  not  yet  integrated  into  the 
LLKB.  The  WG  delivered  the  final  LLKB  capability  at  the  conclusion  of  Phase  4  in 
January  2003. 


FINDINGS 

The  lessons  learned  WG  examined  the  strengths  and  weaknesses  of  the  various 
database  efforts  and  experienced  very  candid,  insightful  discussions  and  technical 
exchanges  with  the  joint  and  service  lessons  learned  SMEs.  After  an  extensive  survey 
of  current  Service  and  agency  lessons  learned  efforts,  a  general  roadmap  to  objective 
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capability  was  outlined.  Due  to  the  complex  nature  of  database  development  and  the 
intricacies  of  forensic  engineering  requirements  of  the  SAT,  the  implementation  plan 
first  focused  on  delivery  of  a  “first  step”  prototype  database,  which  is  referred  to  as 
Phase  1 .  The  LL  WG  required  extensive  SME  and  Service  liaison  to  scope  the  initial 
data  fields  and  structure.  Once  the  initial  configuration  was  determined,  feedback  and 
modification  requests  were  used  to  scope  follow-on  versions  of  the  LLKB. 


CONCLUSIONS 

The  WG  addressed  the  SIAP  SE  TF,  Service  and  Agency  requirements  by  developing 
an  initial  capability  to  maintain  and  provide  system  capability  information  to  support  joint 
interoperability  planning,  testing,  and  forensics  engineering.  Building  on  this  first  step, 
the  WG  continued  to  implement  the  four-phased  implementation  of  the  LLKB,  producing 
a  classified  SIAP  lessons  learned  and  knowledge  repository. 


RECOMMENDATIONS 

The  WG  will  continue  to  solicit  feedback  for  additional  LLKB  capability  from  the  SIAP  SE 
TF  and  Service/Agency  SMEs.  The  WG  will  also  continue  to  integrate  the  LLKB  into 
SIAP  SE  TF  plans,  processes  and  procedures  to  enhance  the  interoperability  and 
warfighting  capabilities  of  joint  and  service  air  warfare  systems. 
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1  INTRODUCTION 

As  stated  in  the  Single  Integrated  Air  Picture  (SIAP)  Analysis  Team  Roles  and 
Responsibilities,  the  SIAP  Lessons  Learned  Knowledge  Base  (LLKB)  is  being 
developed  to  support  the  disciplined  SIAP  system  engineering  process  by  facilitating 
root  cause  analysis  and  forensic  engineering,  as  well  as  serving  as  a  knowledge 
repository  for  SIAP-related  information.  The  LLKB  centralizes  the  collection  of 
documented  problems  and  assessments  from  observed  analysis  events  to  assist  the 
SIAP  System  Engineering  (SE)  Task  Force  (TF)  in  its  efforts  to  improve  the  air  picture. 
The  LLKB  will  assist  the  SIAP  Analysis  Team  (SAT)  in  conducting  forensics  engineering 
throughout  the  various  SIAP  analytical  venues  and  events.  The  SIAP  LLKB  will 
enhance  problem  identification,  tracking  and  resolution  and  supports  results  comparison 
for  SIAP  related  exercises/critical  experiments,  Hardware-in-the-Loop  (HWIL),  Operator- 
in-the-Loop  (OITL),  and  other  Modeling  and  Simulation  (M&S)  activities  for  the  systems 
engineering  decision  making  process. 

The  LLKB  system  enables  the  knowledge  gained  from  past  experience  to  be  applied  to 
current  and  future  endeavors.  Its  intent  is  to  avoid  the  repetition  of  past  failures  and 
mishaps,  as  well  as  the  ability  to  share  observations  and  best  practices.  Through  this 
resource,  the  SIAP  SE  TF  seeks  to  facilitate  the  early  incorporation  of  safety,  reliability, 
maintainability,  and  interoperability  into  the  design  of  air  system  hardware,  software, 
and  procedures. 

2  APPROACH 

The  LLKB  prototype  development  began  with  extensive  SIAP  LL  WG  liaison  with  Joint, 
Service  and  Agency  lessons  learned  database  Subject  Matter  Experts  (SMEs).  There 
are  current  Joint,  Service  and  Agency  efforts  that  have  been  thoroughly  explored  by  the 
LL  WG  before  formulating  the  LLKB  implementation  plan.  Understanding  these  existing 
efforts  contributed  significant  knowledge  and  insight  into  the  SIAP  LLKB  prototype 
development  effort. 


3  SCOPE 

This  Technical  Report  (TR)  outlines  the  processes,  procedures  and  approach  the  SIAP 
LL  Working  Group  (WG)  followed  in  developing  the  LLKB.  This  TR  also  documents  the 
technical  specifications  of  the  prototype  and  outlines  an  approach  for  integrating  this 
capability  into  the  SIAP  SE  TF  process.  Issues  concerning  the  process  for  developing 
and  populating  the  LLKB  are  introduced  but  not  thoroughly  defined  in  this  TR. 
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4  SUMMARY  OF  ACTIVITY 

4.1  Initial  Fact  Finding  Visits 

4.1.1  Issues  Tracking  Board  (ITB)/Missile  Defense  Agency  (MDA) 

Discussions  were  held  with  MDA  representatives  that  resulted  in  a  demonstration  of  the 
ITB  database  developed  for  the  MDA.  The  ITB  is  an  internet-based  application  that 
operates  on  standard  HTML/JavaScript  technologies  for  the  client  GUI  (Graphical  User 
Interface).  The  server  uses  a  Linux  operating  system  and  information  is  stored  in  a 
Sybase  version  database.  The  main  features  of  the  ITB  database  are  that  it  allows 
users  to  log-in,  propose  new  issues,  search  existing  issues  using  a  variety  of  attributes, 
create  reports  of  existing  issues,  and  upload  supporting  documents. 

Concerns  were  raised  by  WG  members  as  to  whether  the  ITB  database  structure  and 
programs  would  be  compatible  with  SIAP  LLKB  objectives:  software  and  licensing 
issues,  compatibility  issues  with  ultimate  LLKB  host,  and  training  issues  with 
developers.  The  reorganization  of  MDA  into  National  Teams  was  also  viewed  as  a 
possible  impediment  to  implementation.  The  WG  determined  that  leveraging  the  ITB 
would  result  in  a  high-risk  proposition  for  SIAP  because  of  these  and  other  issues.  As  a 
result,  LLKB  deliverables  and  schedule  would  likely  not  be  completed  on  time  while 
assessing  these  risks,  awaiting  ITB  liaison  approval,  and  getting  the  necessary 
commitment  for  technical  support. 

4.1.2  Air  Force  Center  for  Knowledge  Sharing  and  Lessons  Learned  (AFCKSLL) 

Fact  finding  liaison  with  AFCKSLL  representatives  primarily  focused  on  their  database 
structure  for  possible  adaption  and  use  in  the  LLKB  effort.  The  AFCKSLL  uses  an 
internet  application  developed  with  standard  HTML/JavaScript  technologies  for  the 
client  Graphical  User  Interface  (GUI).  The  server  uses  a  Windows  NT  4.0  operating 
system  and  Internet  Information  Service  5.  Active  Server  Pages  (ASP),  the  language 
used  for  the  middleware  logic  and  SQL  Server  7,  is  the  database  used  to  store  lessons 
learned  information.  The  main  features  of  the  website  are  that  it  allows  users  to  log-in, 
propose  new  issues,  search  existing  issues  using  a  variety  of  attributes,  create  reports 
of  existing  issues,  and  provides  configuration  control  of  issues  using  sharing  options 
and  email  notification.  The  issues  reside  in  a  working  area  until  the  holder  of  the  issue 
publishes  it  to  the  public  domain.  One  drawback  to  the  database  was  the  lack  of  a  file 
upload  capability. 

The  AFCKSLL  uses  Microsoft  products  for  website  development  and  hosting,  which  the 
SIAP  SE  TF  is  familiar  with,  and  their  database  seemed  to  be  flexible  and  simpler  to 
operate  while  still  providing  a  robust  capability.  The  AFCKSLL  is  also  developing  a  form 
in  MS  Word  with  embedded  XML  tags,  which  will  allow  an  issue  to  be  written  offline  and 
later  uploaded  and  read  directly  into  the  database.  The  embedded  XML  tags  however 
require  that  users  upgrade  to  Office  XP.  The  search  feature  was  easy  to  use  and 
allowed  users  to  “drill  down”  to  the  desired  level.  The  tracking  process  used  to  keep 
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people  in  the  loop  sends  out  emails  to  the  appropriate  points  of  contact  when  a  record  is 
updated.  It  also  provides  for  feedback  and  accountability  of  issues  that  are  being 
worked.  There  was  not  an  elaborate  security  front-end  on  the  classified  site  like  the  ITB 
had  built.  Any  user  on  the  SIPRN  only  has  to  register  and  can  gain  access.  They  also 
have  an  unclassified  site  that  is  a  mirror  image  of  their  SIPRN  site  but  without  any  listing 
of  classified  information.  A  Secret  Internet  Protocol  Router  Network  (SIPRN)  address  is 
provided  as  a  placeholder  for  classified  information. 

4.1 .3  Joint  Center  for  Lessons  Learned  (JCLL) 

The  WG  also  initiated  discussions  with  JCLL,  which  were  very  insightful  into  current 
Joint  Forces  Command  (JFCOM)  lessons  learned  efforts.  Their  system  is  currently 
being  re-developed  and  they  are  also  rewriting  their  processes  for  incorporation  it  into  a 
joint  lessons  learned  plan.  Their  database  has  experienced  delays  because  of  the  need 
to  convert  the  basic  AFCKSLL  structure  from  SQL  Server  to  Oracle.  It  was  also  learned 
that  the  Defense  Information  Services  Agency  (DISA)  might  begin  to  require  classified 
database  sites  on  the  SIPRN  to  have  additional  security  features.  JCLL  relies  on  the 
Joint/Service  communities  to  submit  LL  records  after  an  exercise,  unlike  the  Army, 
which  has  the  personnel  to  record  issues  on  site  as  they  happen.  Analysts  at  JCLL 
then  sort  through  submissions  and  determine  which  records  are  appropriate  for  the 
JCLL  database. 

4.1.4  Capabilities  and  Limitations  (C  &  L)  Database 

Extensive  liaison  with  personnel  from  the  Port  Hueneme  Division  (PHD),  Naval  Surface 
Warfare  Center  (NSWC),  centered  around  the  Navy’s  C  &  L  database  and  possible 
linkage  with  the  SIAP  LLKB.  The  C  &  L  database  supports  Navy  battlegroup 
preparation  by  defining  the  proper  linkages  between  all  of  the  different  systems.  It 
detailed  each  system’s  strengths  and  weaknesses,  as  well  as  identifying  the  “bugs”  that 
arise  when  certain  systems  interact  and  the  known  workarounds.  The  “bugs”  are 
submitted  by  the  respective  systems’  program  office.  The  database  mainly  consists  of  a 
large  number  of  HTML  pages  linked  hierarchically  and  cross-linked.  It  also  provides 
some  links  to  other  supporting  documents.  These  links  also  help  ensure  traceability 
back  to  the  original  sources  for  the  information. 

The  SIAP  SE  TF  is  leveraging  the  C&  L  effort  and  provided  funds  for  PHD  NSWC  to 
begin  development  of  a  C  &  L  database  for  the  SIAP  SE  to  include  all  joint  air  warfare 
systems.  As  a  result  of  this  synergy,  it  was  determined  that  a  strong  linkage  between 
the  SIAP  LLKB  and  the  C  &  L  database  should  be  established  and  maintained.  This 
relationship  has  provided  the  WG  with  valuable  information  and  insight  into  many  areas 
of  the  LLKB  development  and  implementation.  PHD  NSWC  personnel  also  offered  to 
host  the  LLKB  on  their  SIPRN  server  along  side  the  C  &  L  database. 

Upon  completion  of  the  fact-finding  effort,  the  WG  formulated  a  draft  implementation 
plan  for  LLKB  development  and  solicited  pointed  feedback  for  the  Services.  The  WG 
resolved  comments  and  feedback,  revising  implementation  plan  accordingly.  This  effort 
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included  documenting  functional  requirements,  database  interfaces,  and  data  field 
definitions  and  descriptions.  The  WG  socialized  recommendations  to  determine  the 
optimal  technical  specifications  as  well  as  the  process  and  procedures  for  updating 
entries/data  residing  in  the  LLKB. 

4.2  LLKB  Phase  1  -  Desktop  Prototype  Development 

The  lessons  learned  WG  focused  on  a  first  step  towards  the  long-term  goal  of  an  on¬ 
line,  interactive  LLKB.  The  WG  developed  the  Phase  1  LLKB  as  a  stand-alone  desktop 
application  that  allowed  exploration  of  initial  data  structures  and  GUI  features  of  the 
SIAP  LLKB.  The  screens  and  features  of  this  prototype  were  developed  to  match  the 
abilities  that  are  possible  in  an  online  application,  the  ultimate  objective.  The  prototype 
allowed  additions  of  new  lessons  learned  to  a  database,  search  for  lessons  learned 
using  various  criteria,  and  creation  of  simple  reports.  In  general,  the  prototype 
supported  the  priorities  articulated  by  the  Service  and  Agency  representatives  in  a 
survey  of  desired  capabilities  completed  in  April  2002.  The  final  results  of  the  survey 
are  documented  in  Appendix  B. 

The  WG  designed  and  developed  the  database  using  Access  2002.  Based  on  the 
information  requirements  of  the  WG,  tables  and  fields  were  designed  and  created.  For 
each  field  the  name,  data  type,  size,  indexing,  description,  etc.  properties  are  specified 
for  the  application  GUI. 

4.2.1  Prototype  GUI 

The  prototype  GUI  was  developed  using  Access  2002,  taking  advantage  of  its  internal 
Visual  Basic  for  Applications  programming  capabilities.  The  GUI  design  was  based  on 
the  Department  of  Energy’s  Lessons  Learned  website  interface.  This  section  describes 
the  SIAP  LLKB  prototype’s  various  screens  and  controls. 

The  Main  Screen  is  the  entry  point  into  the  SIAP  LLKB  application.  The  Main  Screen 
includes  the  following: 

•  SIAP  LLKB  logo 

•  Input  New  Lesson  button 

•  Browse  Lessons  button 

•  Search  Lessons  button 

•  What’s  New  button 

«  Point  of  Contact  (POC)  button 

•  Links  button 

•  Exit  button 
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•  Disclosure  statement 

•  Version  number  and  date 


Lessons  Learned 
Knowledge  Base 


Prototype  Version  3.0 
17  May  2002 


Input  New  Lesson 
Browse  Lessons 
Search  Lessons 
What's  New? 


Welcome  to  Prototype  ver  3.0  of  the  S!AP  System  Engineering 
Task  Force  Lessons  Learned  Knowledge  Base  (LLKB).  The  SIAP  5E 
will  use  the  LLKB  prototype  as  a  working  application  to  record  and 
store  IADS  performance  lessons  learned  from  selected  events  of 
interest.  The  SIAP  Analysis  Team  will  rely  on  the  LLKB  for  root 
cause  analysis  and  various  aspects  of  forensics  engineering. 
Feecback  and  suggestions  from  the  SIAP  and  LL  communities  are 
welcome  and  appreciated.  The  LLKB  Technical  POC  Is  Mr  Joey 
Wang  at  (703)  602-6441  or 


POC  WarnJ@NAVSEA.NAVY.MIL 

Links  FOR  OFFICIAL  USE  ONLY 

All  information  contained  in  this  LLKB  Prototype  is  approved  for 
distribution  to  the  U.S.  Government  only.  All  materials  are  exempt 
from  public  availability  due  to  the  inclusion  of  specific  behaviors, 
capabilities,  and  concept  of  operations  for  Joint  force  IADS 
platforms  and  sensors.  Transmittal  outside  the  U.S.  Government 
must  hove  prior  approval  of  the  SIAP  SE. 
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Figure  1 .  LLKB  Prototype  -  Main  Screen 

The  Input  New  Lesson  screen  allows  a  user  to  submit  a  new  record  to  the  SIAP  LLKB. 

It  will  also  validate  information  entered  before  creating  a  new  record.  The  Input  New 
Lesson  screen  includes  four  tabs  that  allow  input  for  different  types  of  information  about 
the  lesson.  These  tabs  are  described  below: 


1.  General  Information  -  inputs  for  record  title,  status,  and  lesson  service 

2.  Observation  Data  -  inputs  that  describe  the  observation 

3.  SIAP  Information  -  inputs  for  SIAP  related  fields 

4.  POC  -  inputs  for  associating  a  point  of  contact  with  the  record 

A  Submit  Button  validates  the  input  information  and  creates  a  new  record  if  valid.  A 
Close  Button  closes  screen  and  returns  focus  to  the  Main  Screen. 


The  Browse  Lessons  screen  (see  Figure  1)  allows  the  user  to  view  an  entire  record. 
From  this  screen  the  user  can  edit  the  inputs  for  the  currently  viewed  record.  He  can 
also  view  the  next  record,  previous  record,  first  record,  or  last  record  in  the  database. 

The  Search  Lessons  screen  allows  the  user  to  conduct  a  query  for  a  particular  set  of 
records  based  on  user-selected  criteria.  Users  can  search  based  on:  status,  priority, 
exercise/event  name,  exercise/event  date,  and  word  search  on  all  text  fields.  The 
following  buttons  are  provided  to  do  specific  actions: 
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•  Search  -  View  Records  button  -  Conducts  the  search  and  presents  the  results 
using  the  Browse  Lessons  screen. 

•  Search  -  View  Report  button  -  Conducts  the  search  and  presents  the  results 
using  an  Access  report. 

•  Clear  button  -  Clears  the  criteria  input  boxes. 


EE 
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SIAP  LLKB  G3 


Input  New  Lesson  Search 


POC 


Status  :|open  TJ 


Owner  of  Raw  Data:  jwar  fighting  Capability  Shortfalls,  00-024 

Service  of  the  Issue:  (Navy  ~  3 


Record  Number: 


Exercise/Event  Name:  UCET  00 


Observer  Service:  (Army 

List  of  Systems  Involved  in  the  Exercise/Event: 


;»)  Observation  Date:  J  7/1/00  Observation  Time:  |  Zone:  [eST  ~»~j 

’  )  Observer  System:  ^-2C  ;».j 


Exercise/Event  Description: 


|SSDSMk2 

Link/ Architecture  Configuration: 


Observation  Description: 


AEGIS  incorrectly  transmits  Drop  Track  reports  for  tracks  for  which  it  does  not  have  R2.  The  data  link  standards  only  allow  the  R2  unit  to  drop  a 
track. 


Observation  Impact: 


Record:  *<  !  •  lT  ^  V  >  I  Hp/l  of  8 

mm&SmmarnmMSMi  f  IS  Sillteil® 


J 
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Figure  2.  LLKB  Prototype  -  Browse  Lessons  Screen 


The  What’s  New  screen  allows  the  user  to  conduct  a  query  for  recent  lessons  learned. 
This  date  search  is  based  on  the  entry  date  for  the  lesson.  The  following  buttons  are 
provided  to  do  specific  actions: 

•  Search  -  View  Records  button  -  Conducts  the  search  and  presents  the  results 
using  the  Browse  Lessons  screen. 

•  Search  -  View  Report  button  -  Conducts  the  search  and  presents  the  results 
using  an  Access  report. 

•  Clear  button  -  Clears  the  criteria  input  boxes. 

The  POC  screen  allows  the  user  to  view  and  search  for  a  point  of  contact  stored  in  the 
LLKB.  The  “Show  POC  List”  button  produces  a  report  of  all  POC  stored  in  the  LLKB. 

The  Links  screen  contains  a  list  of  hyperlinks  to  various  web  sites  that  may  be  of 
interest  to  the  SIAP  community. 
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On  all  of  the  screens  there  is  a  menu  bar  of  buttons  across  the  top  that  gives  access  to 
the  other  screens  in  the  LLKB. 

4.2.2  Prototype  Documentation 

A  Database  Dictionary,  created  in  Microsoft  Word  format,  was  provided  documenting 
the  database’s  design  and  functionality. 

4.2.3  Prototype  Testing 

The  WG  completed  testing  at  various  steps  along  the  life  cycle  of  the  project.  The  SIAP 
SE  TF  approved  the  resulting  data  structure.  The  WG  members  made  the  overall 
design  and  feature  list  available  to  the  SIAP  SE  TF  members,  as  well  as  service  and 
agency  representatives,  by  placing  a  copy  on  the  official  SIAP  website. 

4.2.4  Training 

The  developers  held  a  training  session  for  the  initial  users  to  educate  on  the  use  of  the 
SIAP  LLKB  prototype.  Suggestions  and  comments  were  solicited  and  many  were 
incorporated  into  the  design  requirements  for  follow  on  phases. 

4.3  LLKB  Phase  2  -  Unclassified  Website 

The  Phase  2  LLKB  website  application  leveraged  the  data  structures  and  GUI  features 
of  the  prototype,  as  well  as  additional  capabilities  and  features,  and  presented  them  in  a 
web-based  environment.  The  Phase  2  website  provided  for  the  quick  addition  of  new 
records  to  the  database  on  line,  a  search  for  lessons  learned  using  various  criteria,  and 
is  able  to  create  simple  reports  of  the  records  contained  in  the  database.  The  Phase  2 
LLKB  also  contained  a  page  of  links  to  various  other  Service  and  Agency  lessons 
learned  sites  of  interest. 

The  Phase  2  website  was  developed  in  standard  HTML  and  JavaScript  to  present  the 
web  interface.  ASP  2.5  is  used  to  interface  with  the  database  on  the  server.  It  provided 
an  opportunity  for  members  of  the  SAT,  Services  and  Agencies  to  review  the  LLKB  and 
provide  feedback  to  the  lessons  learned  WG.  Suggestions,  changes  and  modifications 
to  the  LLKB  were  vetted  through  the  WG  and  approved  modifications  were  incorporated 
into  the  follow-on  LLKB  system. 

The  Phase  2  website  was  also  used  as  part  of  the  “proof  of  process”  opportunity  for  the 
SAT  provided  by  the  Millennium  Challenge  ’02  exercise  held  in  September  2002.  A 
limited  number  of  Test  Observation  Reports  (TORs)  were  selected,  adjudicated  on  site, 
and  entered  into  the  LLKB  to  test  the  feasibility  of  the  proposed  process  documented  in 
section  4.6  for  collection  of  SIAP  lessons  learned  at  live  events. 
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4.4  LLKB  Phase  3  -  SIPRN  Website 

Phase  3  of  the  LLKB  implementation  was  accomplished  with  the  successful  migration  of 
the  unclassified  Phase  2  website  application  on  to  a  SIPRN  host  server.  The  classified 
LLKB  website  contains  all  of  the  Phase  2  capability  with  additional  features  coded  as  a 
result  of  feedback  from  the  unclassified  website.  One  such  additional  capability  is  the 
resource  upload  feature,  which  allows  users  to  upload  electronic  materials  of  interest  to 
a  repository  for  easy  access  by  the  SIAP  community.  These  files  can  also  be  linked  to 
appropriate  lessons  to  give  users  additional  information  regarding  a  specific  exercise, 
event,  or  analysis  effort.  The  Phase  3  LLKB,  like  its  unclassified  predecessor,  also 
contains  a  page  of  links  to  various  other  service  and  agency  lessons  learned  sites  of 
interest.  These  links  are  limited  to  SIPRN  hosted  sites  only  and  are  subject  to  the 
security  regulations  of  each  individual  website.  The  Phase  3  LLKB  also  allows  for  the 
SIAP  SE  TF  to  archive  old  or  outdated  records. 

The  WG  developed  the  requirement  for  additional  website  access  levels  to  restrict 
functions  for  unauthorized  users.  There  are  3  website  access  levels  coded  into  the 
LLKB: 

•  General  Access  -  Any  user  cleared  to  access  a  SIPRN  terminal  can  view  the 
SIAP  LLKB  website.  At  this  level  of  access,  the  user  is  limited  to  browsing  and 
conducting  basic  searches  of  the  LLKB  records. 

•  SIAP/Service  SME  Access  -  Users  are  required  to  use  a  login  and  password  to 
enable  specified  function  above  the  General  access  level.  Some  of  the 
additional  functions  include  the  ability  to  submit  a  lesson  for  review, 
upload/download  resources,  edit  current  lessons,  etc.  The  login  and  password 
for  this  level  of  access  will  be  provided  only  upon  approval  by  the  SIAP  SE  TF. 

•  SIAP  SE  TF  Access  -  Users  are  required  to  provide  a  login  and  password  to 
access  this  level  of  LLKB  functionality.  The  SIAP  SE  TF  can  access  all  functions 
of  the  lower  tiers  and  additionally  are  given  administrator  privileges  to  approve 
submitted  lessons  and  resources,  edit  lists  and  drop-down  tables,  change 
passwords,  archive  and  delete  records,  etc.  This  administrator-level  access  will 
be  reserved  for  SIAP  SE  TF  personnel  only  in  order  to  maintain  configuration 
control  of  the  website  and  database. 

4.5  LLKB  Phase  4  -  Additional  Capability 

The  WG  allowed  for  the  LLKB  to  be  used  and  tested  during  Phase  4  of  the  LLKB 
implementation.  This  enabled  the  WG  to  vet  and  incorporate  suggested  changes  to  the 
LLKB  while  continuing  to  give  the  website  additional  capabilities.  Any  suggested 
changes  or  modifications  to  the  LLKB  may  be  submitted  to  the  LL  WG  chair  for 
consideration.  If  approved  by  the  WG,  post-Phase  4  changes  requested  to  the  LLKB 
will  be  incorporated  as  time  and  resources  allow.  A  software  design  document  was 
delivered  upon  the  completion  of  Phase  4  that  describes  the  technical  specification  and 
functionality  of  the  objective  SIAP  LLKB  system.  This  document  will  be  updated  as 
changed  are  coded  into  the  LLKB  design  throughout  it’s  life  cycle. 
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4.6  LLKB  Support  of  the  SIAP  Analysis  Team  (SAT) 

The  process  by  which  observed  Link  and  systems  deficiencies  and  problems  will  be 
considered  for  lesson  learned  candidate  status  and  analysis  is  delineated  in  the 
following  process  diagram  and  explained  in  this  section.  The  LLKB  assists  the  SAT  in 
forensics  engineering  endeavors  to  find  the  root  cause(s)  of  observed  deficiencies  and 
problems  and  document  findings. 


Figure  3.  Proposed  SIAP  Lessons  Learned  Process 
4.6.1  Exercise  /  Event 

One  of  the  goals  of  the  Millennium  Challenge  02  (MC  02)  was  to  establish  a  process  for 
Joint  Interoperability  Test  Center  (JITC),  Joint  Combat  Identification  Evaluation  Test 
(JCIET),  and  the  SAT  to  conduct  observation  recording,  tracking,  and  adjudication.  In 
parallel,  SIAP  used  the  venue  as  a  test  scenario  for  the  Lessons  Learned  process  at 
SIAP  sponsored  events.  Due  to  limited  funding  and  resources  for  MC  02,  the  complete 
TOR  process  (generating,  exchanging,  tracking  and  adjudicating)  discussed  in  the  MC 
02  Data  Management  and  Analysis  Plan  (DMAP)  was  conducted  to  a  limited  degree 
(i.e.,  a  handful  of  selected  TORs)  to  demonstrate  and  assess  the  observation  collection, 
adjudication,  and  entry  process.  This  is  a  notional  process  that  will  be  matured  in  the 
weeks  after  MC  02  to  meet  JCIET,  JITC,  and  SAT  requirements  for  future  events.  The 
TOR  process  will  be  leveraged  in  order  to  provide  input  to  the  lessons  learned  process. 
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JITC,  JCIET,  and  system  representatives  will  discuss  and  adjudicate  all  TOR  and 
Formal  Analysis  Report  (FAR)  analysis  results.  Once  the  TOR/FARs  are  fully 
adjudicated,  JITC  will  create  a  synopsis  of  the  anomaly  and  analysis  based  on  the 
information  contained  in  the  TORs  and  FARs  and  subsequent  discussions.  JITC,  in 
close  coordination  with  JCIET,  will  include  a  MIL-STD  conformance  section  within  its 
report  for  MC  02. 

After  calculation  of  the  attributes,  JCIET  will  lead  an  effort  to  determine  the  root  causes 
of  SIAP  deficiencies.  For  example,  for  instances  when  the  picture  is  not  common  at  all 
systems,  the  underlying  reasons  for  these  commonality  deficiencies  will  be  determined. 
To  do  this,  tracks  available  to  the  operator  for  surveillance  systems  in  the  data  network 
will  be  matched  to  Time  Space  Position  Indicator  (TSPI).  The  results  of  this  match  will 
assign  tracks  held  by  systems  to  one  or  more  TSPI  aircraft.  Comparisons  between 
systems  and  between  a  system  and  TSPI  can  then  be  made  in  accordance  with  SIAP 
SE  defined  attributes. 

4.6.2  On-site  and  Near  Term  Activities 

SIAP  Critical  Experiments  will  be  tracked  on-site  at  the  event  as  well  as  during  the  root- 
cause  phase  of  the  analysis.  TORs  related  to  Integrated  Air  Defense  System  (IADS) 
performance  assessment  will  be  reviewed  and  adjudicated  during  and  after  the  event  as 
discussed  in  the  MC  ’02  DMAP.  JITC  and  SAT  representatives  will  coordinate  to 
develop  a  schedule  for  adjudicating  TORs  and  identifying  SIAP-related  lessons  learned 
candidates  to  the  SIAP  SE  TF  for  inclusion  in  the  SIAP  LLKB. 

4.6.3  Post-Event  analysis  Process 

After  the  lessons  learned  candidates  have  been  adjudicated  and  entered  into  the  LLKB, 
updating  responsibility  for  each  record  will  be  given  to  the  appropriate  SME  or  SIAP 
point  of  contact.  The  POC  for  each  record  is  responsible  for  updating  designated 
candidate  records  through  the  post  analysis  process  including  Block  related  activities, 
forensics  engineering  /  root  cause  analysis,  and  the  interpretation  of  analysis  results. 
After  sufficient  analysis  is  performed  on  the  candidate,  it  will  be  binned  into  one  of  the 
following  categories  for  resolution: 

•  “Bugs”  -  specific  system  related  issues  usually  caused  by  a  failure  to  properly 
implement  a  requirement.  “Bugs”  are  returned  to  the  respective  program  offices 
for  action. 

•  “Structural”  -  root-cause  issues  shared  by  two  or  more  systems  usually  caused 
by  improperly  derived  or  omitted  operational  requirements.  “Structural”  Theater 
Air  Warfare  issues  are  forwarded  to  the  SIAP  SE  TF  for  prioritization,  engineering 
analysis,  and  solution  recommendations. 

•  “Tactics,  Techniques,  and  Procedures  (TTPs)”  -  issues  identifying  human  error 
or  faulty  procedures  as  a  possible  source  for  the  deficiency,  including  training. 
Clearly  defined  TTP  issues  must  be  returned  to  the  warfighting  community  (i.e., 
JFCOM  for  action). 
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•  “Non-repeatable”  -  these  are  issues  not  seen  by  anyone  else  or  after  review; 
they  are  impossible  to  replicate  or  analyze  due  to  missing  or  wrong  information. 


5  FINDINGS 

From  the  information  gathered  from  the  discussions  with  service  and  agency  lessons 
learned  SMEs,  the  WG  advocated  the  implementation  of  the  LLKB  in  four  distinct  but 
integrated  phases  to  mitigate  scheduling  and  technical  risk.  Phase  1  developed  a 
prototype  desktop  database  capability.  Phase  2  produced  an  unclassified,  on-line 
version  of  the  knowledge  base  based  upon  the  Phase  1  product.  Phase  3  migrated  the 
Phase  2  website,  along  with  WG  approved  modifications,  onto  the  SIPRN.  Phase  4 
sought  additional  feedback  and  added  desired  capabilities  not  yet  integrated  into  the 
LLKB.  The  operational  SIAP  LLKB  was  delivered  upon  completion  of  Phase  4  for  SIAP 
SE  TF  utilization. 


6  CONCLUSION 

The  classified  LLKB  is  currently  hosted  at  PHD  NSWC  due  to  the  established 
relationship  between  the  SIAP  SE  TF  and  PHD  NSWC  on  the  Caps  and  Lims  effort. 
PHD  NSWC  states  that  LLKB  hosting  and  routine  server  maintenance  is  within  the 
current  scope  of  the  current  SOW  with  the  SIAP  SE  TF.  The  WG  will  investigate  the 
possibility  of  hosting  the  LLKB  at  the  SIAP  SE  TF  once  their  SIPRN  connections  and 
terminals  are  installed  and  operational  in  calendar  year  2003. 

The  classified  LLKB  will  assist  the  SIAP  SE  TF  in  its  mission  to  identify  interoperability 
deficiencies  and  solutions  to  air  picture  systems,  resulting  in  an  enhanced  warfighting 
capability  for  joint  forces.  It  will  also  serve  to  inform  the  greater  SIAP  community  of 
known  air  picture  deficiencies  and  possible  solutions  where  known. 


7  RECOMMENDATION 

The  WG  will  continue  to  integrate  the  LLKB  into  the  SIAP  SE  TF  processes  and 
procedures  to  document  air  system  interoperability  problems  and  solutions.  The  WG 
will  also  solicit  user  feedback  from  SIAP  SE  TF,  Service,  and  Agency  SMEs  to  make 
appropriate  modifications  to  the  LLKB  as  needed.  This  feedback  will  serve  to  adjust  the 
LLKB  to  best  serve  the  needs  of  the  SIAP  SE  TF  and  the  greater  SIAP  community. 
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APPENDIX  A:  Acronym  List 
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SIAP  Lessons  Learned  Acronym  List 


AFCKSLL 

ASP 

DMAP 

FAR 

GUI 

HWIL 

IADS 

ITB 

JCIET 

JCLL 

JTIC 

LLKB 

MC 

MDA 

MIL-STD 

M&S 

OITL 

PHD  NSWC 
POC 
SAT 
SE  TF 

-GET-  — ~ . 

SIAP 

SIPRN 

SME 

TOR 

TPSI 

TR 

USJFCOM 

WG 


Air  Force  Center  for  Knowledge  Sharing  and  Lessons  Learned 

Active  Server  Pages 

Data  Management  and  Analysis  Plan 

Formal  Analysis  Report 

Graphical  User  Interface 

Hardware-in-the-Loop 

Integrated  Air  Defense  System 

Issues  Tracking  Board 

Joint  Combat  Identification  Evaluation  Test 

Joint  Center  for  Lessons  Learned 

Joint  Interoperability  Test  Center 

Lessons  Learned  Knowledge  Base 

Millennium  Challenge 

Missile  Defense  Agency 

Military  Standard 

Modeling  and  Simulation 

Operator-in-the-Loop 

Port  Hueneme  Division,  Naval  Surface  Warfare  Center 
Point  of  Contact 
SIAP  Analysis  Team 
System  Engineering  Task  Force 
-Systems  Engineering  Team  ' 

Single  Integrated  Air  Picture 

Secret  Internet  Protocol  Router  Network 

Subject  Matter  Expert 

Test  Observation  Report 

Time  Space  Position  Indicator 

Technical  Report 

United  States  Joint  Forces  Command 
Working  Group 
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APPENDIX  B:  Lessons  Learned  Working  Group  Requirements  Questionnaire 

Summary 
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Requirements  Questionnaire  Summary 


Air  Force 

USMC 

MDA 

NAVSEA  PHD 

Army 

Totals  | 

1 .  The  capability  will  be  hosted  on 
SIPRNET  {NIPRNET  presence  is 
optional  and  requires  additional  effort  to 
administer  and  maintain). 

TRUE 

TRUE 

TRUE 

FALSE 

3  TRUE,  1  FALSE 

2.  The  capability  must  be  in  place  (on¬ 
line  in  SIPRNET)  NLT  Dec  02  -  Jan  03. 

TRUE 

TRUE 

TRUE 

TRUE 

4  TRUE 

3.  Funding  for  the  administration  and 
maintenance  of  the  capability  will  only  be 
available  the  first  two  years  after  going  on 
line.  It  is  SIAP's  intent  to  transfer  this 
responsibility  to  the  hosting 
faciiity/aqency. 

TRUE 

TRUE 

TRUE 

TRUE 

4  TRUE 

4.  Any  changes/modifications  to  the 
capability  will  be  approved  by  the  SIAP 
Analysis  Team  SG  first,  then  provided  to 
the  hosting  facility  for  final  approval  and 
implementation  (as  required).  Originator 
of  chanqe  is  responsible  for  funding. 

TRUE 

TRUE 

TRUE 

TRUE 

4  TRUE 

5.  Full  new  development  requires  high 
programmer  and  web  designer  efforts  vs. 
modifying  existing  capability  by 
application  support  contractors. 

TRUE 

TRUE 

TRUE 

TRUE 

4  TRUE 

6.  An  existing  capability  is  less  flexible 
and  requires  more  coordination  to  adopt 
chanqes. 

FALSE 

TRUE 

TRUE 

TRUE 

3  TRUE,  1  FALSE 

7.  Of  the  two  options  considered  here,  a 
new  development  will  most  likely  re-use 
most  of  the  sub-products  (i.e.  GUI,  data 
structure)  developed  for  the  prototype. 
The  GUI  can  be  reused  by  both  options. 

TRUE 

TRUE 

FALSE 

TRUE 

3  TRUE,  1  FALSE 

8.  Selecting  an  existing  capability  may 
imply  technical/proprietary  restrictions  if 
modifications  are  required 

TRUE 

TRUE 

TRUE 

TRUE 

4  TRUE 

9.  A  new  development  with  proper 
documentation  will  be  easy  to  rehost  as 
needed.  Do  not  necessarily  require  long¬ 
term  support  by  the  developer. 

TRUE 

TRUE 

FALSE 

TRUE 

3  TRUE,  1  FALSE 

10,  Lower  cost  has  greater  importance 
than  shorter  schedule  than  functionality. 

TRUE 

TRUE 

FALSE 

FALSE 

3  TRUE,  1  FALSE 

1.  Data  Entry:  The  ability  to  personally 
update  records  pertaining  to  my  system 
on  line. 

Required 

Required 

Very  Important 

Required 

Required 

19 

3.8 

2.  Data  Structure:  Database  structure 
must  coincide  with  systems  currently 
used  to  document  similar  functions 

Required 

Very  Important 

Important 

Not  Required 

Not  Required 

11 

2.2 

3.  On-Line  Data  Entry:  Ability  to  enter 
data  from  any  remote  site. 

Very  Important 

Important 

Very  Important 

Required 

Very  Important 

15 

3 

4.  Availability:  System  availability  24 
hours  a  day,  7  days  a  week. 

Very  Important 

Very  Important 

Very  Important 

Required 

Important 

15 

3 

5.  Data  Mining:  ability  to  link  or  retrieve 
information  from  other  databases  or  web 
sites 

Required 

Important 

Important 

Very  Important 

Not  Required 

12 

2.4 

6.  Off-line  Data  Entry:  Ability  to  do  data 
entry  offline  and  upload  at  a  future  time. 

Required 

Important 

Important 

Very  Important 

Important 

13 

2.6 

7.  Vetting:  Ability  to  review  and 
comment  on  items  in  the  process  and 
review  cycle  on-line. 

Very  Important 

Required 

Very  important 

Required 

Very  Important 

17 

3.4 

8:  Help  Desk:  Direct  access  to  user 
support  desk  on  As-Required  bases 
during  normal  business  hours. 

Required 

Very  Important 

Important 

Important 

Important 

13 

2.6 

9.  Data  Archiving:  On-line  ability  to 
access  and  recovery  data  up  to  1  year 
old 

Required 

Required 

Very  Important 

Not  Required 

important 

14 

2.8 

10,  Menu  Driven:  Pull  down  menus 

Very  Important 

Very  Important 

Very  Important 

Important 

Very  Important 

14 

2.8 
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UNCLASSIFIED 

DRAFT 

Requirements  Ranking 


ARMY 

NAVY 

USMC 

USAF 

MDA 

Summary 

1 

New  Order 
by  Rank 

Question 

Rank 

wataai 

Rank 

wigmni 

Rank 

Rank 

EEBEE 

Rank 

iwsiffiai 

1 

7 

5% 

7 

4% 

10 

i% 

1 

5% 

4 

15% 

5.83 

6 

2 

1 

25% 

1 

35% 

5 

4% 

8 

15% 

1 

20% 

3.36 

2 

3 

3 

15% 

4 

5% 

2 

21% 

4 

10% 

2 

20% 

3.10 

1 

4 

8 

2% 

8 

4% 

9 

1% 

2 

5% 

10 

5% 

7.42 

8 

5 

9 

1% 

5 

4% 

4 

17% 

10 

15% 

6 

5% 

6.87 

7 

6 

4 

15% 

6 

4% 

6 

4% 

3 

10% 

5 

5% 

4.87 

5 

7 

6 

6% 

10 

0% 

8 

1% 

7 

10% 

8 

5% 

7.83 

10 

8 

10 

1% 

2 

25% 

7 

3% 

9 

15% 

9 

5% 

7.49 

9 

9 

5 

10% 

3 

15% 

1 

33% 

5 

5% 

7 

5% 

4.33 

3 

10 

2 

20% 

9 

4% 

3 

15% 

6 

10% 

3 

15% 

4.70 

4 

100% 

100% 

100% 

100% 

100% 
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